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ABSTRACT 


AvantGuard1  studies  the  interplay  of  human  direction  and  autonomous  behavior  in  conducting  a  simulated  mission. 

It  is  a  computer  game  in  which  Unmanned  Air  Vehicles  (UAVs)  are  used  to  protect  a  convoy  moving  through 
unsecured  urban  terrain  under  conditions  of  asymmetric  conflict.  Adversaries  hide  among  the  residents  and  prepare  an 
ambush.  The  operator  directs  small  UAVs  and  studies  the  resulting  sensor  stream  to  find  the  ambush  before  the  convoy 
arrives  in  the  trap. 

AvantGuard  serves  those  who  seek  to  improve  the  effectiveness  of  the  UAV  mission.  It  is  an  instrument  with  which 
researchers  can  measure  performance  as  they  develop  new  systems. 

It  is  particularly  designed  to  study  the  interplay  of  human  supervisor  and  autonomous  UAVs.  Its  cognitive  challenges 
are  organized  into  distinct  tasks.  For  each  task,  the  autonomy  level  of  the  UAV  is  set  independently.  Calibrated  to  . 
independent  standards,  results  can  be  compared  to  one  another  and  to  the  findings  of  other  researchers. 

By  addressing  real-world  problems,  such  as  battlefield  constraints  on  bandwidth  and  the  limits  of  machine  vision, 
AvantGuard  presents  a  credible  approach  to  mission  simulation,  training  and  eventual  execution.  By  employing 
sophisticated  game  techniques,  AvantGuard  advances  an  innovative  design.  By  considering  the  post-combat  role  of  the 
military,  it  prepares  an  instrument  to  advance  the  goals  of  peace  as  well  as  those  of  war. 
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1  AvantGuard  (Contract  FA8650-04-M-6485)  has  been  funded  by  the  Air  Force  Research  Lab,  Human  Effectiveness 
Directorate  as  a  response  to  SBIR  AF04-071:  Adaptive  Levels  of  Automation  for  UAV  Supervisory  Control 


OVERVIEW 


1  The  Goals  of  the  Project 

The  principle  goal  of  this  project  is  to  maximize  the  effectiveness  of  those  airmen  who  direct  groups  of  unmanned  air 
vehicles.  Effectiveness  is  increased  when  the  human  attention  is  cleared  of  concerns  that  can  be  delegated  profitably  to 
machines  and  is  provided  with  awareness  of  situation  and  consequence  which  is  clear,  deep  and  easily  navigable. 

This  project  exists  to  help  Air  Force  scientists  discover  the  principles  by  which  autonomous  entities  can  assume  certain 
responsibilities  without  compromising  human  intent  and  by  which  they  can  focus  human  attention  without  distorting  the 
sensed  world. 

There  are  two  minor  goals  to  which  this  project  also  commits.  These  goals  can  be  achieved  without  detracting  from  the 
primary  goal.  Rather,  simultaneous  pursuit  of  all  three  goals  seems  likely  to  enhance  the  likelihood  of  success  for  each 
of  them. 

The  secondary  goal  is  to  explore  tactics  for  a  particular  application  of  UAVs.  AvantGuard  studies  an  urban  asymmetric 
confrontation.  It  proposes  a  particular  arrangement  of  sensor-bearing  UAVs,  highly  interactive  control  system,  convoy 
routing  protocols  and  a  trained  airman.  Autonomy  is  distributed  complexly  throughout  this  system.  By  envisioning  and 
testing  this  system,  the  developers  examine  a  new  approach  to  a  serious  wartime  problem. 

When  this  problem  is  extrapolated  forward  into  a  post-combat  environment,  it  is  seen  that  the  challenges  of  peace  can 
exceed  those  of  war.  Existing  concepts  must  deepen  and  all  the  intelligences  must  become  more  subtle.  The  ambition  of 
this  project  does  not  extend  to  solving  these  challenges  -  only  to  elucidate  them. 

The  third  stated  goal  of  AvantGuard  is  to  demonstrate  the  effectiveness  of  a  user  interface  whose  roots  lie  in  videogame 
development  -  rather  than  Human  Factors  engineering.  This  DNA  will  be  obvious  in  the  visuals  it  presents,  in  the 
controls  it  affords  and  in  the  comfort  it  assumes  that  the  human  has  with  artificial  entities  who  display  independent 
behavior. 

1.1  Scope 

Ambitious  goals  can  inspire  fine  achievement.  But,  ill-managed,  they  court  failure.  Stepwise  approach  to  meeting  these 
goals  is  essential  to  project  stability  and  success.  The  most  simple  breakdown  is  the  one  enforced  by  Air  Force  funding 
policies:  a  Phase  I  dedicated  to  proof  of  concept  and  a  Phase  II  which  results  in  a  functional  prototype. 

1.1.1  Phase  I 

Phase  I  aims  to  communicate  the  experimental  concept.  Vigorous  software  development  effort  renders  the  proposed 
user  experience  in  a  tangible  and  visible  form  This  example  supports  an  experimental  plan  to  address  rigorously  the 
issues  of  autonomy. 

1.1.2  Phase  II 

The  experimental  plan  has  been  formalized  as  a  Phase  II  proposal.  That  plan  details  the  structure  of  the  completed 
AvantGuard  software  and  the  scientific  inquiry  for  which  it  is  designed. 

1.1.3  Documentation 

Both  the  existing  software  achievements  and  the  final  product  blueprint  are  summarized  in  this  report.  Interested  readers 
are  referred  to  the  Final  Summary  Video2  for  a  focused  view  of  the  proposed  user  experience. 


2  Available  for  download  at  http://GamesThatWork.com 
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2  Human  and  Artificial  Intelligence 

AvantGuard  isolates  a  problem  and  presents  a  treatment  which  employs  both  autonomous  UAVs  and  human  judgment. 
It  is  a  meaningful  mission  -  low,  slow  reconnaissance  in  an  urban  environment,  protecting  a  moving  convoy. 

Resources  are  limited:  UAV  locations,  sensor  bandwidth,  and  time  remaining  before  convoy  arrival.  But  the  most 
critical  resource  is  the  single  human  intelligence.  No  machine  vision  can  distinguish  threats  in  this  complex 
environment,  where  a  shovel  might  be  more  ominous  than  a  grenade  launcher.  This  project  aims  to  focus  human 
intelligence  on  critical  judgments  by  assigning  other  decisions  to  autonomous  entities. 

3  Issues  of  Autonomy 

The  immediate  aim  of  this  project  is  a  functional  and  flexible  testbed  to  address  the  principles  of  autonomy  design  at  the 
Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate  (AFRL/HE).  The  testbed  must  deliver  valid  scientific 
results  that  are  reproducible  both  in  that  lab  and  elsewhere. 

3.1  Autonomy  has  value 

3.1.1  Reducing  warfighters’  exposure  to  danger. 

No-one  doubts  the  courage  of  American  pilots.  But  there  are  many  missions  in  which  their  commitment  would  be 
squandered.  Some  tasks  present  tedium  beyond  human  tolerance.  Some  have  very  high  risk  but  offer  only  modest 
rewards.  Some  flights  can  have  no  return.  UAV  technology  enables  these  previously  impossible  missions. 

3.1.2  Enabling  innovation. 

Without  the  burden  of  supporting  a  crew,  the  UAV  escapes  many  design  constraints.  It  needs  no  cockpit.  It  can  exceed 
human-tolerable  G  forces.  Or  it  can  be  six  inches  long.  UAV  technology  enables  these  previously  impossible  designs. 

3.1.3  Eliminating  costs. 

Safety,  comfort  and  life  support  systems  can  be  stripped  from  the  design.  Weight  is  reduced,  maintenance  is  easier. 

3.1.4  Decreasing  operational  manpower  requirements. 

Unlike  manned  craft,  the  crew  of  a  UAV  can  be  less  than  one.  However  this  is  not  true  of  all  UAVs.  Although 
remotely-piloted  UAVs  reduce  danger  and  eliminate  costs,  in  operation  they  require  no  less  attention  than  manned  craft. 
The  Predator  has  a  (remote)  crew  of  2.  To  realize  a  manpower  savings,  engineers  must  introduce  autonomy  into  UAVs. 

3.2  But  autonomy  brings  problems. 

3.2.1  Judgment  vs.  Heuristics 

Important  decisions  often  require  high-level  pattern  recognition  or  sophisticated  inference.  Frequently  they  require 
creative  thought.  They  may  demand  a  complex  synthesis  of  disparate  factors.  Humans  perform  such  reasoning  better 
than  computers.  When  such  decisions  are  consequential,  designers  will  temper  UAV  autonomy  with  human  supervision. 

3.2.2  Automation  compromises  performance 

Researchers  in  this  field  have  named  many  of  the  vulnerable  points  where  automation  corrupts  competent  operators: 

•  Automation  bias  [Skitka  2000] 

Operator  reasoning  is  impaired  by  excess  credence  given  to  the  machine’s  suggestions. 

•  Complacency  [Knapp  1991] 

Operator  unconsciously  neglects  attention  to  work  shared  with  automation. 

•  Workload  Imbalance  [Yeh2001] 

Automation  distorts  the  job,  forcing  greater  attention  to  lesser  tasks. 

•  Skill  Degradation  [Parasuraman  2000] 

Human  skills  which  are  performed  by  automation  fall  out  of  practice. 


AvantGuard:  An  instrument  to  explore  Adaptive  Levels  of  Autonomy 


3 


•  Over-reliance  [Wickens  1999] 

Operator  consciously  denies  responsibility  for  automated  tasks. 

•  Attention  Narrowing  [Mosier  1998] 

Decision-making  is  impaired  by  misdirected  attention  or  by  filtered  sensor  data. 

•  Mistrust 

Unwarranted  bias  to  discount  machine  decisions,  often  based  on  an  atypical  experience. 

•  Impaired  vigilance  [Molloy  1996] 

A  species  of  reliance  exacerbated  by  boredom,  that  increases  over  time. 


3.3  Autonomy  design  has  many  dimensions 

3.3.1  Autonomy  has  many  levels. 

UAV  autonomy  demands  interplay  between  man  and  machine.  It  is  this  interplay  -  not  AI  algorithms  -  that  lies  at  the 
heart  of  Level  of  Autonomy  design.  The  levels  are  marked  by  differences  in  notification,  consent,  and  exception,  and 
commonly  parsed  into  linear  scales  of  automation,  such  as  the  ten-level  scales  proposed  by  Sheridan  [1983],  [2000]. 

3.3.2  Autonomy  has  many  domains. 

Each  decision  has  several  cognitive  steps.  These  have  clear  dependencies,  but  they  do  not  necessarily  occur  in  order.  By 
breaking  behavior  into  these  domains  of  cognition,  autonomy  design  for  each  domain  can  be  considered  individually. 

3.3.3  Autonomy  has  many  phases. 

The  pattern  of  autonomy  levels  is  not  static.  It  must  adapt.  For  example:  autonomous  re-planning  might  be  inappropriate 
during  typical  mission  execution,  but  might  be  a  necessity  during  an  episode  of  catastrophic  overload. 

3.4  Tools  for  Autonomy  Design 

Design  of  a  system  with  so  many  variables  requires  guidance.  Researchers  support  designers  by  proving  principles  and 
providing  guidelines.  To  do  so,  the  researchers  require  good  tools.  AvantGuard  is  such  a  tool.  It  presents  a  timely  and 
relevant  task  in  a  compelling  videogame  format.  The  difficulty  of  the  task  is  open-ended:  as  automation  (or  skill 
development)  reduces  the  task  pressure,  more  work  load  can  be  readily  added  by  the  researcher. 

The  cognitive  demands  of  the  game  divide  cleanly  into  five  distinct  tasks.  Each  is  unambiguously  identified  within  a 
widely  accepted  system  of  categorization.  For  each,  the  researcher  sets  a  Level  of  Autonomy  (LoA).  The  settings  refer 
to  an  array  of  distinct  behaviors,  arranged  on  an  ordinal  scale,  calibrated  to  an  independent  standard. 

Finally  it  offers  the  researcher  several  modalities  for  adaptive  Level  of  Autonomy.  Broad  standards  for  Adaptive  LoA 
(ALoA)  have  yet  to  be  established.  (Perhaps  some  will  emerge  from  this  tool.)  The  tool  supplies  certain  ‘natural’ 
algorithms  for  Adaptive  Level  of  Autonomy.  Also  it  invites  creativity:  A  Finite  State  Machine  offers  the  researcher  a 
simple  and  flexible  approach  to  original  ALoA  strategies.  A  user-friendly  tool  allows  non-programmers  to  compose 
such  strategies. 
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4  Game  Techniques 

Even  if  game  interface  conventions  had  no  inherent  superiority,  they  would  be  worth  adopting:  Big  gains  in 
performance  and  in  learning  curve  reduction  are  available  by  simply  exploiting  the  existing  highly  developed  skills  of  a 
gamer  generation.  Maintain  a  videogame  sensibility,  and  tap  into  that  training. 

But  Hollywood  game  techniques  are  not  arbitrary.  They  are  highly  evolved  designs  that  optimize  situation  awareness, 
reaction  time  and  engagement.  AvantGuard  explores  a  variety  of  techniques  familiar  to  gamers. 

4.1  Cinematography 

Attention  is  directed  -  and  situation  awareness  is  sharpened  -  by  dramatic  staging,  framing  and  lighting. 

Rich  models  of  realistic  scenes  increase  verisimilitude  and  engagement.  Simplicity  clarifies  symbolic  scenes  involving 
cartographic,  diagrammatic,  abstract  data  representation. 

4.2  Camera 

Camera  moves  add  situation  awareness.  They  display  three-dimensional  relationships  better  than  a  static  viewpoint  can. 

4.3  Challenges: 

Well  designed  challenges  sharpen  attention  -  rather  than  distract. 

Visible  score  maintains  focus  on  goal.  “Personal  Best”  is  an  eternal  challenge. 

4.4  Control 

Complete  control  of  point-of-view  and  moving  through  sensor  stream  database. 

Two  handed  (keyboard  and  mouse)  control,  or  traditional  game  controller:  fast,  accurate,  known. 

4.5  Customization 

AvantGuard,  like  most  complex  modem  computer  games,  facilitates  customization  (Game  Mods)  by  its  users. 

The  best  of  these  modifications  improve  the  fidelity  of  the  scenarios  by  incorporating  actual  experience. 

4.6  Community 

Multiplayer  networks  allow  users  to  share  learning  within  the  game  itself,  through  conversation  or  observation. 

Online  exchange  of  Mods  -  posted  on  websites  or  exchanged  through  email  allow  rapid  emergence  of  quality. 

4.7  Competition 

Performance  is  enhanced  by  competition  for  scores.  Creative  solutions  emerge  from  competition  for  mod  popularity. 
Competitive  antagonistic  play  brings  out  better  performance  on  both  sides,  far  better  than  artificial  intelligence. 

4.8  Composition 

There  is  a  potential  for  musical  elements  to  convey  certain  elements  of  awareness  as  they  do  in  popular  games. 

Music  arguably  reduces  fatigue  and  boredom. 

4.9  Creative  Effects 

Visuals  effects  stimulate  and  refresh  the  operator.  They  can  emphasize  elements  that  require  special  attention. 

Sound  effects  and  haptics  add  more  dimensions  of  sensory  involvement. 

4.10  Conversation 

Speech  Recognition  is  used  (starting  in  Phase  I)  to  reroute  the  convoy. 

Interactive  Voice  Response  (Phase  II)  add  depth  to  the  convoy  routing  phase  by  simulating  human  conversation. 

It  has  the  potential  to  speech-enable  other  parts  of  the  human-machine  dialogue. 
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AVANTGUARD 


5  Three  Tier  Architecture 

The  experimental  system  has  a  straightforward  architecture.  Three  networked  processes  manage  different  functionality. 
Phase  I  resulted  in  a  broad  functional  specification  of  two  of  these  processes:  Database  of  Experiments  (DBX)  and 
Experimenter’s  User  Interface  (XUI).  The  third  process,  the  AvantGuard  Game  Interface,  has  been  developed  in  far 
more  detail. 


Y|  |*ri  Experimenter's 
AUli  User  Interface 


AVANT  GUARD: 


Subject’s 
Game  Interface 
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5.1  DBX:  Database  of  Experiments 

A  network  server  hosts  a  Relational  Database  Management  System.  This  holds  the  entire  history  of  the  experiments: 
each  study  is  characterized  by  an  Experiment  Design  and  by  the  large  body  of  results.  These  results  can  be  examined  in 
detail  or  in  aggregate  using  a  standard  query  language,  SQL.  AFRL  researchers  can  easily  perform  statistical  analyses 
on  the  simple  results  of  the  experiments.  Careful  database  design  facilitates  creative  data-mining.  The  same  results  can 
be  re-explored  using  fresh  models  -  and  previously  unanticipated  correlations  can  be  discovered. 

Finally  a  networked  server  and  a  clearly  designed  SQL  database  offers  investigators  the  ability  to  publish  raw  data. 

Peers  can  verify  the  conclusions  of  the  authors.  Fresh  minds  can  interpret  these  results  from  a  new  perspective.  A  ready- 
for-public  relational  database  management  system  will  allow  investigators  to  publish  data  easily  in  the  widely  accessible 
XML  (Extensible  Markup  Language)  format. 
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5.2  XUI:  Experimenter’s  User  Interface 

The  researcher  will  typically  not  work  at  a  game  machine,  nor  the  database  server,  but  at  an  ordinary  office  computer, 
possibly  remotely.  The  Experimenter’s  User  Interface  affords  the  researcher  the  big  picture  -  and  the  small  details  of 
the  study.  The  XUI  is  an  online  interface  to  the  database  server  accessible  with  a  simple  web  browser.  It  is  presented  on 
a  web  client,  with  the  application  running  on  the  server  as  middleware  logic,  interfaced  locally  with  the  SQL  server. 

5.2.1  XUI  for  the  Experiment  Designer 

The  experimental  design  package  allows  the  Principal  Investigator  to  establish  the  scope  of  a  study.  The  sample  size, 
controls,  independent  and  dependent  variables  are  identified.  Experiment  design  methodologies  such  as  factorial, 
fractional,  and  nested  design  will  be  supported.  The  product  of  this  designer’s  effort  is  recorded  in  the  database,  not 
only  as  study  parameters,  but  as  a  new  relational  table  structure,  designed  to  capture  these  specific  experimental  results. 

5.2.2  XUI  for  the  Research  Manager 

Administrative  tools  of  the  XUI  are  concerned  with  operational  aspects  of  an  experimental  run.  It  includes  the  interface 
with  which  individual  subjects  are  enrolled,  debriefed  and  tracked.  Management  of  the  sample  population  is  built  in.  For 
example,  blind  blocked  randomization  of  subjects  into  control  and  experimental  groups. 

5.2.3  XUI  for  the  Session  Monitor 

During  actual  experimental  session,  the  XUI  registers  the  subject  and  launches  the  appropriate  session.  As  the  session  is 
executed,  it  records  performance  metrics  of  the  subject. 

5.2.4  XUI  for  the  Scientist 

Finally  the  XUI  can  be  used  to  extract  and  present  results.  Simple  metrics  can  be  queried  individually,  or  in  the 
aggregate.  Results  can  be  displayed  graphically  -  in  a  format  conducive  to  public  presentation. 

The  XUI  offers  limited  statistics  and  graphics  for  quickly  browsing  experimental  results.  It  will  not  attempt  to  reproduce 
a  full  statistics  package.  But  neither  does  it  limit  inquiry.  Researchers  can  compose  any  SQL  expression  in  the  interface 
and  download  the  results  in  XML  format.  Unlimited  statistical  analysis  can  be  performed  on  these  data  with  an 
appropriate  mathematics  package. 

5.3  AvantGuard:  The  Game  Interface 

The  AvantGuard  application  runs  well  in  standalone  mode  for  testing  and  development.  Before  actual  experimentation, 
it  is  configured  by  parameters  (such  as  Level  of  Automation  points)  in  the  database.  Detailed  designs  of  the  urban 
landscape  and  its  threats  are  also  loaded  from  repositories  on  the  server.  During  the  experimental  session,  data  streams 
from  this  client  to  the  server,  where  it  is  recorded  -  and  optionally  formatted  for  real-time  display  to  the  researcher.  At 
the  end  of  the  session,  summary  tables  are  developed  in  the  database. 

There  is  another  use  for  this  high  performance  “game  computer”.  Researchers  will  use  it  when  developing  a  set  of 
challenges  for  an  experimental  session.  AvantGuard  provides  a  complete  “level  editor”  with  which  a  researcher  can 
place  and  program  a  wide  variety  of  humans,  vehicles  and  weapons.  The  resulting  design  is  stored  on  the  database 
server.  During  experimental  design,  the  researcher  chooses  among  these  scenarios.  The  active  scenario  design  is 
archived  by  the  server  and  remains  available  even  if  the  file  is  later  lost  or  modified. 
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6  Autonomy  Controls 


6.1  Parasuraman  Domains 

Parasuraman’s  four  domains  of  cognition  form  a  useful  tool  for  organizing  the  issues  of  autonomy. 

Reduced  to  monosyllables,  these  are:  Look,  See,  Choose,  Act. 

We  have  organized  the  cognitive  tasks  of  AvantGuard  into  the  four  Parasuraman  Domains.  The  tasks  are  actually 
divided  into  five  distinct  processes,  with  two  different  Domain  1  tasks,  one  creative  and  one  technical.  These  are  listed 
in  the  table  below  and  explained  in  detail  in  the  following  pages. 

6.2  Sheridan  Levels 

Each  of  the  tasks  has  a  strategy  to  implement  all  ten  of  the  Sheridan  [2000]  levels  of  automation.  These  had  to  be 
artfully  composed.  All  strategies  fit  the  definitions,  but  each  of  the 
tasks  needs  a  distinct  set  of  strategies. 

6.3  Adaptive  Levels  of  Autonomy 

AvantGuard  seeks  adaptive  algorithms  that  adjust  autonomy  levels 
in  response  to  events.  Events  include  mission  phase  change, 
operator  behavior,  pop-up  threats,  equipment  malfunction,  etc. 

Certain  algorithms  are  recognized  now.  More  will  added  during  the 
to  design  original  algorithms. 


Parasuraman 

AvantGuard 

Intel.  Acquisition 

Site  selection 

Intel.  Analysis 

Photo  examination 

Decision 

Threat  Assessment 

Execution 

Convoy  reroute 

Fig  6.1  AvantGuard 

tasks  in  Parasuraman  space  | 

next  two  years,  as  will  a  facility  for  the  researcher 


6.3.1  Initial  Algorithms  for  Adaptive  Levels  of  Autonomy 

6. 3. 1.1  Crisis  Driven 

Begin  at  LoA  2,  (very  little  automation.) 

If  Subject  does  nothing,  switch  to  LoA  6  (management  by  exception)  just 
before  disaster  strikes. 


6 

£ 

4 

3 

2< 

l- 


Crisis  Drived 


6. 3 . 1 . 2  Oversight  Atrophy 

Begin  at  LoA  5  (management  by  consent) 

If  subject  accepts  nothing-  elevate  to  LoA  6 

If  Subject  vetoes  nothing,  elevate  to  LoA  7  (act  and  report  action) 

If  Subject  corrects  nothing,  elevate  to  LoA  8  (report  only  on  request) 
If  Subject  asks  to  see  action,  revert  to  LoA  5 


assertion 

Iff 

c  . .  , 

1 

* 

4  .  _  . 

Oversight  Atrophy 

6. 3. 1.3  Drill  Down 

Begin  at  LoA  6  (management  by  exception) 

If  Subject  raises  an  exception,  descend  immediately  to  LoA  5 

If  Subject  continues  to  assert  control  by  tweaking  suggested  action,  descend 

to  LoA  2. 
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6.3.2  User  Defined  Adaptively 

The  researcher  can  design  a  pattern  of  behavior  which  implements  an  original  strategy  for  adaptive  Level  of  Autonomy. 
The  pattern  is  defined  as  a  series  of  responses  to  various  events.  A  Finite  State  Machine  can  manage  these  responses  by 
treating  them  as  individual  state  transitions  which  are  easily  and  unambiguously  encoded  as  truth  table  logic. 

Researchers  may  avoid  working  with  raw  finite  state  machines.  GamesThatWork  has  begun  an  exploratory  cooperation 
with  Stottler  Henke,  a  company  that  has  developed  a  Behavior  Editor  for  the  USAF.  Rather  than  introduce  a  new  way  to 
specify  AI  algorithms  to  Air  Force  researchers,  it  may  be  possible  to  incorporate  Stottler  Henke’s  SimBionic  engine, 
which  will  allow  use  of  the  behavior  editor  which  is  designed  for  intelligent  non-programmers  to  use  and  has  adherents 
through  the  USAF  [Stottler], 


H 

User  requests  suggestion 

3 

7 

•j 

3 

4 

,!*n 

User  accepts  suggestion 

3 

4 

n5 

:•  ::-7V 

User  makes  choice  from  option  list 

2 

Tweak  Node  by  Node 

i 

2 

2 

2 

3 

3 

Time  Out  (no  user  response) 

2 

3 

4 

5 

6 

7 

Reopen  and  Change  Routing  Decision 

i 

2 

3 

3 

4 

4 

4 

4 

4 

4 

Failure*  due  to  Inaction 

3 

4 

5 

6 

6 

Failure*  due  to  Action 

3 

3 

4 

4 

5 

6 

7 

8 

9 

10 

*  "Failure”  event  occurs  when  convoy  enters  an  active  threat  area. 

Fig  6.  3.2:  Finite  State  Machine  truth  table  used  to  program  LoA  adaption 

This  table  encapsulates  the  transition  rules  for  a  finite  state  machine. 

Each  row  of  the  table  represents  a  particular  event  during  the  execution  of  the 
experiment.  The  events  include  both  user  actions  and  internal  simulation 
results.  (Only  a  few  examples  are  presented  here.) 

Each  column  represents  a  Level  of  Autonomy  at  which  the  simulation  can  be 
working  when  the  event  occurs. 

Each  cell  in  the  table  shows  the  new  Level  of  Autonomy  which  results  when  the 
event  occurs  at  that  level.  Grayed  cells  reflect  the  fact  that  all  events  cannot 
occur  at  all  Levels  of  Autonomy. 

For  example,  if  the  User  requests  a  suggestion  from  a  UAV  at  LoA  1  or  2,  the 
UAV  will  adaptively  switch  to  LoA  3.  If  asked  again  for  a  suggestion,  it  will 
remain  at  LoA  3. 


6.3.3  Adaptability  Designed  In 

AvantGuard’s  simplified  arrangement  of  tasks  may  be  still  too  complex  for  certain  studies.  Five  tasks  might  be  too 
many  moving  parts.  For  example,  the  current  design  specifies  two  actions  in  the  Parasuraman  first  domain:  (Site 
selection  and  Flight  planning).  An  experimenter  might  choose  to  eliminate  the  latter,  fairly  mechanical  step  and  remove 
it  cleanly  from  the  subject’s  vision.  By  setting  the  task  to  LoA  10,  the  task  disappears  -performed  automatically  and 
never  presented  to  the  subject. 
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7  Five  Tasks  in  AVANT  GUARD 


7.1  SITE  SELECTION 


7.1.1  Task: 

The  operator  knows  the  convoy  route  and  the  past  history  of  attacks  in  this  area.  The  operator  must  choose  those  sites 
which  require  coverage  and  determine  from  which  angles  they  should  be  seen.  Specifically  the  operator  marks  the  areas 
to  be  photographed  by  the  UAVs.  These  can  be  particular  building  faces,  rooftops,  intersections  or  entire  street  fronts. 
The  operator  has  a  map  of  the  town.  This  is  a  full  three-dimensional  (3D)  map,  including  terrain  and  individual  building 
elevations,  which  AvantGuard  can  render  in  cartographic  or  videogame  mode. 

7.1.2  Interface: 

The  operator  has  both  map  and  3D  model  and  an  instant,  smooth,  automated  transition  between  the  two.  The  operator 
can  select  sites  by  example  -  by  pointing  the  ‘camera’  (first-person  point  of  view)  at  any  scene  in  the  3D  model,  and 
clicking  a  shot.  A  UAV  will  fly  to  this  location  and  camera  orientation  and  send  back  a  photograph. 

The  Operator  can  also  indicate  directly  what  is  to  be  photographed,  by  clicking  on  it.  Power  tools  allow  one  to  choose  a 
street  easily  -  with  or  without  the  facing  building  fronts.  A  very  precise  degree  of  control  is  available  to  an  operator 
who  wants  to  select  exactly  the  angles  and  distances  from  which  the  site  should  be  seen. 

7.1.3  Par  asuraman  Domain  1 : 

Site  Selection,  along  with  Flight  Planning,  is  part  of  “Look”  (intelligence  acquisition).  Of  the  two,  it  is  the  creative  task: 
determining  the  dangerous  points  along  the  convoy  route. 

7.1.4  Autonomy  Levels: 

At  the  Level  1,  the  operator  selects  each  shot  in  teach-by-example  mode.  At  higher  levels  the  system  selects  coverage 
along  the  route.  Some  levels  call  for  operator  coverage  choice,  street-by-street  tweaks,  or  plan  confirmations.  The 
highest  do  not  require  this.  At  Level  10,  of  course,  the  task  disappears  into  full  automation. 

7.1.5  Error: 

AvantGuard  must  accurately  model  not  only  functional  autonomy  -  but  also  errors  in  autonomy.  Site  selection  errors 
are  simple:  False  negative  means  the  omission  of  suspicious  sites.  False  positives  include  both  marking  harmless  sites 
and  dangerous  sites  that  are  off  the  route. 


AvantGuard:  An  instrument  to  explore  Adaptive  Levels  of  Autonomy 


10 


7.2  FLIGHT  PLANNING 


Fig  6.2b:  Waypoint 


7.2.1  Task: 

The  operator  designs  the  flightpath  of  each  UAV.  The  object  is  to  efficiently  fly  by  the  surveillance  sites.  The  plan  must 
maximize  the  number  of  sites  visited,  and  must  also  time  them  properly.  The  most  useful  photo  is  one  taken  just  before 
it  is  examined.  The  most  meaningful  time  to  examine  the  photo  is  at  the  last  possible  moment  -  just  in  time  to  detour  the 
convoy  if  needed. 

7.2.2  Interface: 

Flightpaths  are  set  by  establishing  waypoints.  Each  UAV  (or  formation  of  UAVs)  flies  a  curve-fit  path  (Hermite  spline) 
along  the  sequence  of  waypoints.  The  operator  is  afforded  both  a  top-down  map  interface  and  full  3D  flight  planner. 

The  top  down  map  view  is  well  suited  to  large  scale  overall  flightpath  decisions: 

•  assign  photo  op  coverage  to  particular  UAVs 

•  choose  order  of  coverage  by  each  UAV 

•  deconflict  UAV  flightpaths 

•  arrange  overlap  and  multi-angle  coverage 
The  3D  view  allows  the  operator  to 

•  optimize  viewing  angle,  especially  for  urban  scene 

•  avoid  terrain  and  obstacles  in  low  altitude  flight 

•  establish  clear  lines  of  sight 

In  either  view  waypoints  can  be  established  -  and  dragged  anywhere.  The  waypoint  altitude  can  also  be  easily  adjusted  - 
either  with  a  slider  in  the  properties  window  or  by  dragging  with  a  mouse  button  chord. 

7.2.3  Parasuraman  Domain  1: 

Flight  Planning,  is  also  part  of  “Look”  (intelligence  acquisition).  Unlike  Site  Selection,  the  job  of  directing  UAVs  to 
cover  a  set  of  sites  has  an  optimal  solution.  Computer  algorithms  have  no  disadvantage  in  addressing  this  task. 

7.2.4  Autonomy  Levels 

At  the  Level  1,  the  operator  sets  each  waypoint.  Level  2  provides  ‘wizard’.  Each  subsequent  level  offers  a  more  robust 
routing  solution  using  established  algorithmic  approaches  to  the  Traveling  Salesman  Problem,  (TSP)  the  classic,  well- 
studied  puzzle  of  computational  mathematics. 

7.2.5  Error: 

AvantGuard  models  autonomy  errors  by  photographing  sites  from  poor  angles  and  distances.  It  can  produce  inefficient 
paths.  And  it  exacts  a  high  cost  in  processing  time,  delaying  the  responsiveness  of  the  autonomous  system. 
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7.3  PHOTO  EXAMINATION 


Fig  6.3a:  Sensor  image  stream  -  embedded  in  scene.  Fig  6.3b:  Image  examination  with  high-resolution  inset. 


7.3.1  Task: 

The  operator  examines  incoming  sensor  data  for  indicators  of  threats.  The  operator  looks  for  characters  in  threatening 
poses:  holding  or  hiding  weapons,  ducking  behind  walls  and  parapets,  suspicious  use  of  binoculars,  cell  phones,  or 
batteries. 

The  operator  reviews  low  resolution  pictures  downloaded  from  the  UAVs.  Bandwidth  constraints  require  that  high 
resolution  imagery  is  downloaded  from  the  on-board  UAV  cache  only  on  operator  request.  The  operator  selects 
subsections  (“chips’)  within  the  photographs  to  acquire  high-resolution  detail.  The  operator  must  balance  the  need  for 
detail  with  the  bandwidth  cost  and  must  act  before  the  cached  image  on  the  UAV  has  been  discarded. 

7.3.2  Interface: 

The  operator  picks  photos  directly  from  the  3D  scene.  They  are  embedded  in  the  scene  corresponding  to  where  they 
were  shot.  Once  chosen,  photos  sweep  into  flat  view  floating  over  the  symbolic  3D  view.  High  resolution  chips  are 
downloaded  by  simply  dragging  out  areas  within  the  image. 

A  toolbar  on  the  bottom  offers  the  red,  yellow,  green  judgment  buttons.  It  also  has  a  suite  of  tools  to  allow  the  operator 
to  navigate  spatially  and  temporally  through  the  image  database. 

7.3.3  Parasuraman  Domain  2: 

Analysis  of  sensor  intelligence  -  such  as  examination  of  surveillance  photos,  is  Parasuraman’s  second  domain  (See). 

7.3.4  Autonomy  Levels: 

At  Level  1,  the  operator  has  only  simple  tools  for  image  management.  Higher  levels  order  the  photos  intelligently.  Tree 
autonomy  begins  with  Automatic  Target  Cueing  (ATC).  This  begins  by  directing  the  operator’s  attention  and  progresses 
to  filtering  the  image  stream.  This  advances  into  more  assertive  systems  of  Automatic  Target  Recognition  (ATR). 

7.3.5  Error: 

Autonomy  here  is  problematic:  The  science  of  machine  vision  is  imperfect.  The  consequence  of  error  can  be  grave.  But 
most  of  all,  the  entire  problem  is  difficult.  Locating  humans  is  not  the  problem.  The  goal  is  to  distinguish  threats  from 
neutrals.  A  man  in  a  threatening  pose  is  easily  recognized  by  a  human,  but  not  by  a  machine.  And  context  is  important. 

A  shovel  is  generally  innocuous,  but  in  certain  contexts  it  is  more  damning  than  a  Kalashnikov. 

Simulated  errors  in  ATC  must  be  accurate.  The  system  cannot  offer  a  random  point  in  a  random  photograph.  It  must 
select  man-like  imagery,  and  other  images  that  closely  resemble  known  threat  patterns. 
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7.4  THREAT  ASSESSMENT 


7.4.1  Task: 

The  operator  identifies  active  threats  and  makes  a  judgment  on  the  safety  of  a  street  segment  (one  block  or  intersection). 
The  operator  bases  this  judgment  on  suspicious  photos  of  this  area.  If  they  indicate  danger,  the  operator  marks  the 
individual  street  segment  as  a  threat.  These  accumulate  to  form  the  Threat  Map.  The  decision  usually  requires  the 
operator  to  integrate  the  content  of  several  photos,  perhaps  the  same  area  viewed  from  different  angles  or  monitored 
over  a  period  of  time. 

7.4.2  Interface: 

The  operator  works  primarily  in  an  overhead  view.  Footprints  of  the  photographs  are  projected  onto  the  map,  allowing 
the  operator  to  readily  locate  red-flagged  photographs.  Clicking  on  a  footprint  displays  the  photograph  in  context.  The 
operator  sees  the  image  from  the  point  of  view  of  the  sensor  that  acquired  it. 

The  normal  image  examination  tools  are  available,  but  of  limited  value.  The  operator  can  ‘look  through”  the  picture  at 
the  model  beneath  it.  By  looking  at  all  the  overlapping  suspect  photographs,  the  operator  evaluates  and  precisely  locates 
the  threat.  If  it  is  determined  that  the  Threat  is  severe,  a  Threat  Marker  is  placed  on  the  map. 

7.4.3  Parasuraman  Domain  3: 

Threat  Assessment  falls  in  the  third  domain  (Choose).  A  pattern  of  danger  has  been  established  and  a  judgment  must  be 
formed  on  its  significance. 

7.4.4  Autonomy  Levels: 

Autonomy  can  provide  reasonable  threat  determinations  since  the  photographs  themselves  have  already  been  examined 
and  judged.  A  variety  of  algorithms  (among  them  fuzzy  logic)  suggest  themselves,  and  the  levels  of  autonomy  simply 
involve  shifting  authority  from  the  human  operator  to  these  algorithms. 

7.4.5  Error: 

The  errors  that  these  algorithms  produce  will  not  be  hard  to  simulate.  These  will  result  from  poorly  biased 
discrimination  -  either  excess  or  insufficient  caution. 
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7.5  ROUTE  SELECTION 


7.5.1  Task: 

The  operator  maps  the  best  street-by-street  route  based  on  avoiding  threats.  Secondary  considerations  in  the  routing 
decision  include  the  incident  history  (avoid  neighborhoods  that  have  been  proven  dangerous)  and  the  characteristics  of 
the  street  and  structures  (prefer  broad  avenues  and  low  buildings). 

7.5.2  Interface: 

The  operator  will  find  it  easiest  to  work  in  the  Map  View  -  the  two-dimensional,  top-down.  North-up  perspective.  The 
most  natural  interface  for  a  route  decision  is  an  ordinary  flat  map,  and  it  will  feel  familiar.  However  an  AvantGuard 
innovation  -  the  ‘Flywheel3’device  allows  the  operator  to  instantly  swoop  the  point  of  view  to  any  angle,  so  that  lines  of 
sight  and  vantage  points  can  be  readily  assessed.  Just  as  easily,  the  Map  view  can  be  restored. 

While  making  Routing  decision,  the  convoy  position  and  its  destination  are  visually  prominent.  The  streets  are  easily 
seen.  A  series  of  street  segments  are  highlighted.  These  show  the  current  convoy  route.  Some  segments  are  clear,  and 
some  are  blocked  by  Threats  that  require  the  convoy  to  detour.  Incident  history  is  visible,  but  somewhat  less  prominent. 

When  the  mouse  hovers  over  the  route,  it  intensifies  in  brightness.  Routing  “handles”  appear  at  each  intersection.  The 
operator  can  click  drag  a  handle  to  a  new  intersection.  The  convoy  reroutes  efficiently  through  the  new  intersection.  The 
criterion  used  to  calculate  efficiency  depends  on  the  selected  Level  of  Autonomy. 

7.5.3  Parasuraman  Domain  4 : 

The  execution  of  the  decision  is  to  reroute  the  convoy.  This  is  Level  4  (Act). 

7.5.4  Autonomy  Levels 

Simple  point-to-point  routing  can  be  accomplished  with  great  reliability  using  Dykstra  techniques.  Accurate  graphs  and 
properly  weighted  costs  result  in  optimal  paths.  Sheridan’s  autonomy  levels  are  implemented  by  this  algorithm. 

7.5.5  Error: 

In  addition  to  stochastic  false  positives  and  false  negatives,  the  system  has  a  more  interesting  failure.  Veterans  of  Iraq 
report  encountering  vivid  danger  signs  (such  as  a  burning  vehicle)  that  served  only  to  detour  a  cautious  convoy  directly 
into  the  real  ambush.  Such  scenarios  can  be  set  up  in  AvantGuard,  and  will  usually  trap  the  autonomous  systems. 


3  The  Flywheel  device  is  a  user  interface  construct,  designed  around  the  mouse  middle  button  and  wheel.  It  allows  the  user  to  grasp 
any  point  on  an  object  or  terrain  and  to  ‘fly’  relative  to  that  point  in  three-space.  An  unanchored  ‘free  flight’  mode  also  exists. 
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BEYOND  PHASE  II 


The  initial  challenge  is  simply  to  achieve  the  goals  of  Phase  II  and  to  realize  this  existing  design.  Phase  II  projects 
twenty  four  months  of  development.  During  those  two  years  -  and  afterward  -  AvantGuard  must  remain  relevant  to  the 
challenges  of  the  real  world.  It  must  anticipate  change. 

Today  AvantGuard  models  active  urban  insurgency.  Only  a  grim  pessimist  would  predict  that  American  forces  will  still 
conduct  open  warfare  in  the  cities  of  Iraq  in  2007.  But  withdrawal  is  also  improbable.  Most  likely,  Americans  will  be 
engaged  as  armed  peacekeepers. 

Peacekeepers  must  fight  differently:  A  force  that  supports  the  rule  of  law  cannot  ignore  the  law.  Battlefield  tactics  that 
might  be  justifiable  today  will  be  unacceptably  brutal  when  the  goal  is  to  stabilize  a  civil  society. 

Just  as  UAVs  in  the  game  scout  ahead  of  the  convoy,  AvantGuard  is  itself  a  look-ahead  mission.  It  aims  to  address  this 
upcoming  role.  Precise  acquisition  of  intelligence  will  be  more  important  than  precise  delivery  of  munitions.  Good 
UAV  surveillance  supports  a  commander’s  search  for  a  wise  plan.  Good  UCAVs  support  only  one  plan. 

In  this  light,  consider  the  fidelity  of  AvantGuard’s  current  scenario.  It  is  not  a  very  accurate  model  of  a  mature 
resistance.  In  a  peacekeeping  situation,  adversaries  will  not  congregate  conveniently  on  roof  tops.  The  cues  of  danger 
will  be  based  on  long  baseline,  high  detail  observations.  Breaks  in  the  pattern  reveal  danger  -  often  through  negative 
observations  of  civilian  behavior  (“Where  are  the  school-kids  today?”,  “Usually  there  are  cars  parked  out  front  here.”). 
AvantGuard  is  extensible.  With  strengthened  archives,  it  can  support  this  patient  patrol  work  and  other  new  missions. 
As  military  roles  evolve,  as  adversaries  are  better  understood,  more  accurate  scenarios  will  be  built. 


CONCLUSION 

Phase  I  expressed  a  commitment  to  three  goals:  the  primary  question  of  autonomy,  the  application  of  autonomous 
vehicles  in  a  convoy  protection  mission  and  the  demonstration  of  videogame  principles  to  controlling  this  application. 
The  work  just  concluded  demonstrates  a  vision  for  achieving  these  goals,  a  plan  for  implementing  the  vision  and  a 
sound  probability  of  success  in  execution  of  the  plan  during  Phase  II. 
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